Consider the kinds of musical behaviours that typical non-musically trained listeners with normal hearing engage in as part of everyday life. Such listeners can tap their foot or otherwise move rhythmically in response to a musical stimulus. They can quickly articulate whether the piece of music is in a familiar style, and whether it is a style they like. If they are familiar with the music, they might be able to identify the composer and/or performers. The listeners can list instruments they hear playing. They can immediately assess stylistic and emotional aspects of the music, including whether or not it is loud, complicated, sad, fast, soothing, or generates a feeling of anxiety. They can also make complicated socio-cultural judgments, such as suggesting a friend who would like the music, or a social occasion for which it is appropriate.
Now, if the listeners are hearing-impaired, what would their musical behaviour be? Partial or profound lack of hearing makes the other ways humans use to sense sound in the environment much more important for the deaf than for people with normal hearing. Sound transmitted through the air and through other physical media such as floors, walls, chairs and machines act on the entire human body, not just the ears, and play an important role in the perception of music and environmental aspects for all people, but in particular for the deaf. In fact, it has been found that some deaf people process vibrations sensed via touch in the part of the brain used by other people for hearing. See D. Shibata “Brains of Deaf People ‘Hear’ Music” in International Arts-Medicine Association Newsletter, 16, 4 (2001). This provides one possible explanation for how deaf musicians can sense music, and how deaf people can enjoy concerts and other musical events.
These findings may suggest that a mechanism to physically ‘feel’ music might provide an experience to a hearing impaired person that is qualitatively similar to the experience a normal hearing person has while listening to music. However, little research has specifically addressed the question of how to optimize a musical experience for a deaf person.
Some previous work has been done on providing awareness of environmental sounds to deaf people. (See F. W. Ho-Ching, et al., “Can you see what I hear? The Design and Evaluation of a Peripheral Sound Display for the Deaf,” in Proceedings of the SIGCHI (Conference on Human Factors in Computing Systems 2003), ACM Press (2003), pgs. 161-168; and T. Matthews, et al., “Visualizing Non-Speech Sounds for the Deaf,” in Proceedings of ASSETS (Proceedings of the 7th International ACM SIGACCESS Conference on Computers and Accessibility 2005), ACM Press (2005), pgs. 52-59.) However, no guidance is available to address the challenges encountered at the early stage of designing a system for the deaf to facilitate a better appreciation of music.
Music and the Deaf
Profoundly deaf musicians and those with less pronounced hearing problems have clearly demonstrated that deafness is not a barrier to musical participation and creativity. Dame Evelyn Glennie is a world renowned percussionist who has been profoundly deaf since the age of 12 years but ‘feels’ the pitch of her concert drums and xylophone, and the flow of a piece of music through different parts of her body—from fingertips to feet. Other examples include profoundly deaf musicians such as Shawn Dale—the first and only person born completely deaf who achieved a top ten hit on Music Television (MTV) in 1987; and Beethoven, the German composer who gradually lost his hearing in mid-life but who continued to compose music by increasingly concentrating on feeling vibrations from his piano forte.
Visualising Music
The visual representation of music has a long and colourful history. In the early 20th century Oskar Fischinger, an animator, created exquisite ‘visual music’ using geometric patterns and shapes choreographed tightly to classical music and jazz. Walt Disney, in 1940, released a movie called ‘Fantasia’ where animation without any dialogue was used to visualise classical music. Another example is Norman McLaren, a Canadian animator and film director who created ‘animated sound’ by hand-drawn interpretations of music for film. (See R. Jones and B. Nevile, “Creating Visual Music in Jitter: Approaches and Techniques,” in Computer Music Journal, 29, 4 (2005) pgs. 55-70.) Among the earliest researchers to use a computer based approach was J. B. Mitroo who in 1979 input musical attributes such as pitch, notes, chords, velocity, loudness, etc., to create colour compositions and moving objects. (See J. B. Mitroo, et al., “Movies from Music: Visualizing Musical Compositions,” in Proceedings of SIGGRAPH 1979 (International Conference on Computer Graphics and Interactive Techniques), ACM Press (1979), pgs. 218-225.) Since then, music visualisation schemes have proliferated to include commercial products like WinAmp® and iTunes®, as well as visualizations to help train singers. It is not the purpose of this work to discuss full history here. B. Evans in “Foundations of a Visual Music,” Computer Music Journal, 29, 4 (2005), pgs. 11-24 gives a review of visual music. However, the effect of these different music visualizations on the hearing impaired has not been scientifically investigated and no prior specific application for this purpose is known to Applicants.
Feeling Music
As mentioned above, feeling sound vibrations through different parts of the body plays an important role in perceiving music, particularly for the deaf. Based on this concept, R. Palmer, in 1994, developed a portable music floor which he called Tac-Tile Sounds Systems (TTSS). However, Applicants have not been able to find a report of any formal objective evaluation of the TTSS. Recently, Kerwin developed a touch pad that enables deaf people to feel music through vibrations sensed by the fingertips. (See “Can you feel it? Speaker Allows Deaf Musicians to Feel Music,” Brunel University Press Release, October 2005.) The author claimed that, when music is played, each of the five finger pads on a device designed for one hand vibrates in a different manner and this enables the wearer to feel the difference between notes, rhythms and instrument combinations. As in the previously cited TTSS by Palmer, not many technical or user test details about this device are available. M. Karam, et al., developed an EmotiChair which transforms an audio signal into discrete vibro-tactile output channels using a Model Human Cochlea (MHC), and these output channels are presented in a logical progression along the back of the body. (See M. Karam, et al., “Modelling Perceptual Elements of Music in a Vibrotactile Display for Deaf Users: A Field Study,” in Proceedings of ACHI, 2009 (Second International Conferences on Advances in Computer-Human Interactions, 2009), pp 249-254; and M. Karam, et al., “Towards a Model Human Cochlea: Sensory Substitution for Crossmodal Audio-Tactile Displays,” in Proceedings of Graphics Interface 2008, Windsor, Ontario, Canada, May 28-30, 2008, pgs. 267-274.) Gunther, et al., introduced the concept of ‘tactile composition’ based on a similar system comprised of thirteen transducers worn against the body with the aim of creating music specifically for tactile display. (See E. Gunther, et al., “Cutaneous Grooves: Composing for the Sense of Touch,” in Proceedings of 2002 Conference on New Instruments for Musical Expression (NIME-02), Dublin, Ireland, May 24-26, 2002, pgs. 1-6.)
The closest commercially available comparisons to Applicants' proposed invention include the ‘Vibrating Bodily Sensation Device’ from Kunyoong IBC Co, the ‘X-chair’ by Ogawa World Berhad, the ‘Multisensory Sound Lag’ (MSL) from Oval Window Audio, and Snoezelen® vibromusic products from Flaghouse, Inc. These devices are designed to process sound, including music inputs according to pre-defined transformations before producing haptic output. The Kunyoong IBC Co's Vibrating Bodily Sensation Device only stimulates the one part of the body (the lower lumbar region of the body which is more sensitive to lower frequencies).